Skip to content

Handle NACK from a Shimmer3/3R instead of dropping the connection - #293

Merged
JongChern merged 7 commits into
masterfrom
DEV-982_nack_handling
Nov 13, 2026
Merged

JongChern merged 7 commits into
masterfrom
DEV-982_nack_handling

Conversation

@marknolan

@marknolan marknolan commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

The protocol has had a NACK_COMMAND_PROCESSED (0xFE) reply for a long time, but
the driver never learned about it. Grepping the whole driver for NACK — excluding
the generated instruction set and the unrelated Verisense protocol — returned two
comments and nothing else.

Since the review (10 Nov)

  • master merged in (8a8a771f, no conflicts); ShimmerDriver 178/178, ShimmerDriverPC 103/103, CI green.
  • The SD-sync claim is corrected below. With SD sync enabled the firmware switches Bluetooth off while idle, so the port never opens on either branch. The refusal gate applies only while a sync recording has the radio on.
  • The idle teardown takes about 5 s, not 2 s. When not streaming, a SET gets ACK_TIMER_DURATION + 3 s before the ACK timer runs out.
  • TestNackNotStreaming is reworked as suggested in the review: it refuses 0x30 (always NACKed, changes nothing) with a GET queued behind it, and gives a verdict only on a connected, idle unit whose firmware refuses 0x30. It was run on hardware: this branch PASS 3/3, master FAIL 3/3 (see Verification).
  • The streaming branch's CRC follow-up (waiting for and consuming the NACK's own CRC) is in DEV-1431: a log start the device refuses (NACK) ends at once, idle or mid-stream #345, which is stacked on this PR, along with in-memory tests of both NACK paths. It goes in after this.

What was happening

Both protocol implementations tested only for ACK, with no else:

  • ShimmerBluetooth.java:833
  • LiteProtocol.java:569

So a NACK was read, silently discarded, and the transaction stalled with
mWaitForAck still true and the instruction stack still locked. About five seconds
later the ACK timer expired (when not streaming, a SET gets ACK_TIMER_DURATION + 3 s)
and checkForAckOrRespTask called connectionLost().

There was no retry to soften it. NUMBER_OF_TX_RETRIES_LIMIT is 0, and reading
the NACK byte itself resets mNumberofTXRetriesCount to 0, so 0 >= 0 always
holds, one of the two connectionLost() branches always won, and the soft-recovery
else was dead code.

A refused command was indistinguishable from a dead link, and cost the
connection.

Where this actually bites

checkForAckOrRespTask branches on whether the device is streaming. The
connectionLost() teardown above is the not-streaming branch; while streaming
the same timeout instead calls clearAllInstructions(), which destroys the queued
instructions but leaves the link up.

Configuring a device mid-stream is not a supported operation — the only commands
sent while sensing are start/stop onboard recording and stop streaming, and
START_LOGGING_ONLY_COMMAND/STOP_LOGGING_ONLY_COMMAND already have their own
special-cased timeout handling in checkForAckOrRespTask that avoids
connectionLost(). So the not-streaming branch is the one that matters, and an
earlier revision of this description had that backwards.

Counting the firmware's refusals in log-and-stream-common/Comms/shimmer_bt_uart.c,
exactly one of ~14 sendNack sites is the sensing gate. The rest are reachable
while idle:

Condition Line
SD sync: everything except SET_SD_SYNC_COMMAND and ACK, but only while a sync recording has the radio on (see below) 839, via ShimBt_isCmdAllowedWhileSdSyncing
a sync command while sync is disabled (same XOR gate) 839
payload truncated past args[] 852
SET_SAMPLING_RATE_COMMAND with a zero clock divider 1156
calibration RAM write out of range 1195
SET_DAUGHTER_CARD_MEM EEPROM write failure 1397
SET_CHARGE_STATUS_LED_COMMAND — dispatched but never implemented 1408
SET_BT_COMMS_BAUD_RATE — not supported 1417
SET_INFOMEM_COMMAND offset out of range 1502
RESET_BT_ERROR_COUNTS 1586, 1589
blocked while sensing (ShimBt_isCmdBlockedWhileSensing) 845 — the only streaming one

Corrected after review: an earlier version of this description said a
sync-enabled Shimmer cannot be connected to on master, because the sync gate refuses
the driver's whole initialisation sequence. That is wrong. With SD sync enabled the
firmware switches Bluetooth off while idle (ShimConfig_checkBtModeFromConfig),
so a host connection never starts, on either branch; the gate only applies while the
sync code has the radio on during a sync recording. The other idle refusals in the
table stand.

The plainest one is SET_CHARGE_STATUS_LED_COMMAND (0x30), which the firmware has
refused unconditionally since Shimmer3R LogAndStream v1.01.013 and Shimmer3
v1.01.004. The driver knows 0x30 as SET_BLINK_LED and still sends it from the
public writeLEDCommand(), which its check-alive task calls as a ping while a stream
has stalled. That is the case the bench test now uses.

The existing coping strategy was avoidance rather than handling, which is decent
evidence the failure mode is real — readPressureCalibrationCoefficients() skips
the BMP581 calibration read with the comment "its firmware NACKs … so don't send
it."
Avoiding each NACK individually does not scale, and it makes every new
firmware NACK a latent connection-drop in every released copy of the driver.

What this changes

A refusal now unwinds the transaction and keeps the connection up.

ShimmerObject adds NACK_COMMAND_PROCESSED = (byte) 0xFE
processNackFromCommand() stops the ACK timer, clears mWaitForAck/mWaitForResponse, reports progress, drops the refused instruction so the stack advances, completes the transaction, releases the stack lock
eventCommandRefused() the application hook, called after the unwind

The refused instruction is dropped only when it is still queued, which
processNackFromCommand() determines from mWaitForAck on entry. A GET's
instruction is removed as soon as its ACK arrives, so an unconditional removal
would drop whatever was queued behind it and silently skip that command —
Copilot caught that in review, and it is fixed in 648897ba. The
ACK-wait and streaming paths both still have the instruction queued; the
response-wait path does not.

Handled in three places:

  1. Waiting for an ACK, not streaming — the main configuration path, and the
    one that matters.
  2. Waiting for a response, not streaming — a NACK can land here instead if the
    firmware only discovers it cannot serve a GET while building the response.
    Checked before isKnownResponse(). LiteProtocol gained the matching branch
    in 648897ba after review; it only had the ACK-wait one at first.
  3. A data packet followed by a NACK while streaming — off the supported path,
    but still strictly better than master destroying the whole queue.

Completing the operation (8bd5b940)

Unwinding the transaction was not enough on its own. Operation completion is driven
off the ACK path: sendProgressReport(...) advances a counter, and
ShimmerDeviceCallbackAdapter only calls finishOperation() once
mProgressCounter == mProgressEndValue. processNackFromCommand() sent no
progress report, so a single refused command inside an operation meant the
operation never completed
— mOperationUnderway stayed true and the application
sat in CONFIGURING/CONNECTING indefinitely.

That matters most at connect time, where every command is part of an operation.
Master never completed the operation either, but it killed the connection first, so
the hang was never visible; making the link survive is what exposes it.

The report is sent before the refused instruction is removed, and only while it is
still queued, because BluetoothProgressReportPerDevice.updateProgress() derives
the counter from the remaining stack size — the response-wait path already reported
when its ACK arrived, and reporting twice there would skip a count. The same three
timer-driven commands the ACK path excludes are excluded here.

Known limitation: a refused command still counts as progress, so the operation
completes as SUCCESS. Reporting a partially-refused operation as
FAIL_WITH_WARNING would mean changing the shared OperationState handling in
finishOperation(), which is wider than this fix; eventCommandRefused() remains
the hook for an application that needs to know.

Two deliberate non-obvious choices

eventCommandRefused() is concrete with an empty default, not abstract — and
not a new ProtocolListener callback either. Every other notification hook here
(sendProgressReport, connectionLost, eventLogAndStreamStatusChanged) is
protected abstract, so adding one would break every downstream subclass and
listener implementation at compile time. This way nothing outside the driver has to
change, and anyone who wants the refusal surfaced overrides it.

A stray NACK with no command in flight is logged and ignored, not treated as a
refusal.
processBytesAvailableAndInstreamSupported() runs only when
!mWaitForAck && !mWaitForResponse, so there is no transaction to fail there. The
TypeScript SDK gates its own NACK framing the same way (on a command genuinely
awaiting a reply) so that a leaked stream byte cannot fabricate a refusal; same
reasoning applied here.

Two supporting changes

  • The resync helper accepted only 0x00 and 0xFF as a packet boundary, so a
    NACK sitting in the buffer could not be skipped past cleanly. It now accepts
    0xFE, and is renamed findOffsetOfNextZeroOrFF → findOffsetOfNextPacketBoundary
    to match what it actually does (private, one call site).
  • NACK is registered in mBtCommandMapOther so btCommandToString names it in
    the logs. Deliberately not in mBtResponseMap — isKnownResponse() must stay
    false for it, or processResponseCommand() would try to parse it as a response
    body.

LiteProtocol gets the same treatment (it is live — BasicShimmerBluetoothManagerPc
instantiates it). Its NACK value is declared locally because the generated
LiteProtocolInstructionSet has no NACK entry; adding it to
LiteProtocolInstructionSet.proto and regenerating would be tidier, but that
rewrites a large generated file committed in two places for the sake of one
constant. Happy to do it that way instead if preferred.

Verification

Bench-tested for both cases.

Mas confirmed on real hardware that master never recognises a device NACK (it times
out silently) while this branch detects it, unwinds and advances. That run used the
SET-while-sensing case, so it exercised the streaming branch — which is why the
idle connection drop did not reproduce in it, and why a command queued behind the
refused one did not survive on master: the streaming timeout calls
clearAllInstructions() by design.

TestNackNotStreaming (ShimmerPCBasicExamples, reworked in 7351b7e4) drives the
not-streaming path. Run it with the Shimmer's Bluetooth COM port as its argument, on
an idle Shimmer with SD sync off:

  1. It waits for FULLY_INITIALIZED (20 s). Without it, the verdict is INCONCLUSIVE.
  2. It checks the firmware refuses 0x30 (Shimmer3R v1.01.013+, Shimmer3 v1.01.004+) and the Shimmer is not sensing; otherwise INCONCLUSIVE.
  3. It queues 0x30 0x00 and GET_SAMPLING_RATE_COMMAND behind it, and follows both through the instruction stack for 8 s.
  4. PASS: 0x30 retired, the GET answered, the link up after 8 s. FAIL: the link torn down, or either instruction never completed.

It now lives in com.shimmerresearch.bluetooth, because the instruction list and the
transaction state are protected. Nothing is written to the device.

On a Shimmer3R, LogAndStream v1.01.025, over classic Bluetooth, 3 runs each with the
same test source:

Result Detail
this branch PASS 3/3 0x30 refused and retired after 137-224 ms, the GET answered after 241-272 ms, link up after 8 s, NACK Received for Command: logged
master FAIL 3/3 no NACK logged, the GET never sent, the link lost 5.0 s after the refusal

ShimmerDriver and ShimmerPCBasicExamples both compile clean with no new
warnings under Gradle 8.10, the version CI uses.

The note that used to be here about the Gradle 6.1 wrapper no longer applies: master
now has the 8.14.3 wrapper (DEV-1013), and this branch has master merged in.

Automated tests come with #345, stacked on this PR. ShimmerDriverPC has since
gained an in-memory radio for the receive path (API_00027 onwards), and #345 uses it
to drive both NACK paths: idle through the real IOThread, and streaming through the
parser one byte at a time.

Sibling implementations

The same wire format is implemented independently in several other places, so this was
checked against all of them rather than assumed to be Java-only. Three carried the same
defect and now have their own PRs:

State
Shimmer-C-API same defect #204 - 0xFE falls to the switch default, reported as "Misaligned ByteStream Detected" while streaming, then disconnects after ~10 s of read timeouts
SwiftAPI same defect, worse outcome #18 - the pending CheckedContinuation is never resumed, and timeoutInSeconds is declared but never used, so the caller's await never returns at all
pyshimmer same defect, worse outcome seemoo-lab/pyshimmer#45 - the unhandled byte raises out of _run_readloop, which catches only ReadAbort, killing the reader thread and blocking every waiter for good
shimmer-web-sdk already correct devices/shimmer3/protocol.ts defines NACK = 0xFE, and Shimmer3Client gates the framing on a command genuinely being in flight
Shimmer-MATLAB-ID, Shimmer-Labview-API, Shimmer-Advance-API nothing to do wrappers over the Java or C# drivers, so they inherit the fix

Nothing links these repos and nothing tests them together, so the timing differs wildly
by accident: about five seconds here when idle, ten in C#, never in Swift and Python.

Not in scope

The related writeMem defect from the tracker ticket is untouched here: the
over-range branch in writeMem() detects a buffer running past the device's memory
range and does nothing, its body being entirely commented out, and
MAX_CALIB_DUMP_MAX (4096) disagrees with the firmware's SHIMMER_CALIB_RAM_MAX
(1024 on Shimmer3) so the guard would be wrong even if enabled. Deciding what should
happen on an over-range write is a separate question from teaching the driver to
read a NACK, so I have left it for its own change.

🤖 Generated with Claude Code

…ction

The Shimmer3/Shimmer3R protocol has had a NACK_COMMAND_PROCESSED (0xFE)
reply for a long time, but the driver never learned about it. There was no
NACK constant anywhere on the Shimmer3 side, and both protocol
implementations tested only for ACK with no else branch, so a NACK was read,
silently discarded, and the transaction stalled with mWaitForAck still true
and the instruction stack still locked. Two seconds later the ACK timer
expired and checkForAckOrRespTask called connectionLost(): a refused command
was indistinguishable from a dead link, and there was no retry to soften it
because NUMBER_OF_TX_RETRIES_LIMIT is 0 and reading the NACK byte itself
resets mNumberofTXRetriesCount, making the soft-recovery branch dead code.

This is already reachable on shipping firmware - any SET command while the
device is sensing, a sync-mode mismatch, a failed SET_DAUGHTER_CARD_MEM, and
the BMP180/BMP280 calibration reads on a Shimmer3R all NACK. The existing
coping strategy was avoidance rather than handling; see the comment on
GET_PRESSURE_CALIBRATION_COEFFICIENTS_COMMAND, which is skipped for a BMP581
specifically because the firmware NACKs it.

A refusal now unwinds the transaction and keeps the connection:

- ShimmerObject: add NACK_COMMAND_PROCESSED = (byte) 0xFE.
- ShimmerBluetooth.processNackFromCommand(): stop the ACK timer, clear
  mWaitForAck/mWaitForResponse, drop the refused instruction so the stack
  advances (as both ACK paths do for their own command), complete the
  transaction and release the stack lock.
- Handled in three places: waiting for an ACK while not streaming (the main
  path), waiting for a response while not streaming, and a data packet
  followed by a NACK while streaming, which is the SET-while-sensing case.
- eventCommandRefused() is the application hook. It is deliberately concrete
  with an empty default rather than abstract, and not a new ProtocolListener
  callback, so every existing subclass and listener implementation keeps
  compiling. Override it to surface the refusal.

Two supporting changes:

- The resync helper accepted only 0x00 and 0xFF as a packet boundary, so a
  NACK in the buffer could not be skipped past cleanly. It now accepts 0xFE
  too, and is renamed from findOffsetOfNextZeroOrFF to
  findOffsetOfNextPacketBoundary to match what it does.
- NACK is registered in mBtCommandMapOther so btCommandToString names it in
  the logs. Deliberately not in mBtResponseMap: isKnownResponse() must stay
  false for it, or processResponseCommand() would try to parse it as a
  response body.

A stray NACK arriving with no command in flight is logged and ignored rather
than turned into a refusal, in processBytesAvailableAndInstreamSupported()
which runs only when !mWaitForAck && !mWaitForResponse. The TypeScript SDK
gates its own NACK framing the same way, so a leaked stream byte cannot
fabricate a refusal.

LiteProtocol gets the same treatment. Its NACK value is declared locally
because the generated LiteProtocolInstructionSet has no NACK entry; adding it
to LiteProtocolInstructionSet.proto and regenerating would be tidier but
rewrites a large generated file committed in two places for one constant.

Compile-verified only - there is no test harness for the BT receive path
(ShimmerBluetooth is abstract with many abstract I/O methods and no test
instantiates it), and no hardware was involved.

Co-Authored-By: Claude Opus 5 <[email protected]>
Copilot AI lite review requested due to automatic review settings September 3, 2026 13:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There is at least one confirmed transaction-stack correctness bug (and an uncovered NACK path in LiteProtocol) that can cause queued commands to be dropped or time out incorrectly.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the Shimmer driver to recognize and handle NACK_COMMAND_PROCESSED (0xFE) so a refused command unwinds the in-flight transaction and the connection is not dropped due to an ACK/response timeout.

Changes:

  • Adds a protocol constant for NACK (0xFE) and registers it for logging/command-to-string mapping.
  • Handles NACK in ShimmerBluetooth across ACK-wait, response-wait, and streaming (data packet followed by NACK) paths.
  • Adds analogous NACK handling and an overridable eventCommandRefused(...) hook in LiteProtocol.
File summaries
File Description
ShimmerDriver/src/main/java/com/shimmerresearch/driver/ShimmerObject.java Adds NACK_COMMAND_PROCESSED constant (0xFE).
ShimmerDriver/src/main/java/com/shimmerresearch/comms/radioProtocol/LiteProtocol.java Introduces NACK constant and transaction-unwind handling plus eventCommandRefused(...).
ShimmerDriver/src/main/java/com/shimmerresearch/bluetooth/ShimmerBluetooth.java Adds NACK support in multiple receive paths, updates packet-boundary resync, and adds eventCommandRefused(...).
Review details

Suppressed comments (1)

ShimmerDriver/src/main/java/com/shimmerresearch/bluetooth/ShimmerBluetooth.java:1379

  • findOffsetOfNextPacketBoundary() uses a raw 0 instead of the existing DATA_PACKET constant; now that this method was touched, using the named constant would avoid a magic number and match the surrounding code style (e.g., bufferTemp[0]==DATA_PACKET).
			byte b = buffer[i];
			if (b == 0 || b == ACK_COMMAND_PROCESSED || b == NACK_COMMAND_PROCESSED)
				return i;
  • Files reviewed: 3/3 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…pilot review)

processNackFromCommand() removed instruction 0 unconditionally, but a GET's
instruction is already removed when its ACK arrives. A NACK arriving in the
response-wait state therefore dropped whatever was queued behind it and
silently skipped that command. It is now dropped only when mWaitForAck was
still set, which is exactly the state that still has it queued. Same fix in
LiteProtocol, and the javadoc no longer claims the drop is unconditional.

Also from the review:
- LiteProtocol was missing the response-wait NACK branch that ShimmerBluetooth
  has, so a NACK there would still have fallen through to the timeout. Added,
  checked before isKnownResponseByte() to match.
- The resync helper's inline comment still said it looked for DATA_PACKET or
  ACK only.
- The helper used a raw 0 rather than the DATA_PACKET constant it now sits
  alongside (suppressed review comment).

Co-Authored-By: Claude Opus 5 <[email protected]>
Copilot AI review requested due to automatic review settings September 3, 2026 13:47
@marknolan

Copy link
Copy Markdown
Member Author

All four review items are addressed in 648897ba — the three inline comments have individual replies, and this covers the suppressed one.

Suppressed comment (findOffsetOfNextPacketBoundary using a raw 0): agreed and fixed — it now reads b == DATA_PACKET || b == ACK_COMMAND_PROCESSED || b == NACK_COMMAND_PROCESSED, which is both self-documenting and consistent with bufferTemp[0]==DATA_PACKET in the surrounding code. Worth noting this was the most useful of the four after the transaction-stack bug, despite being the one held back.

Summary of the round:

Item Outcome
processNackFromCommand() dropping the wrong instruction Real bug, fixed. The drop is now conditional on mWaitForAck, which is set only in the states that still have the instruction queued. Fixed in LiteProtocol too, and the javadoc corrected.
LiteProtocol missing the response-wait NACK branch Fixed — added, checked before isKnownResponseByte() to mirror ShimmerBluetooth.
Stale resync inline comment Fixed.
Raw 0 instead of DATA_PACKET (suppressed) Fixed.

The PR description has been updated rather than just corrected here, since the first version described the instruction drop as unconditional and a future reader could otherwise "fix" the code back to that.

Still compile-verified only, and still no automated coverage of the receive path — the transaction-stack bug is a good argument for the fake-transport harness noted in the description, since a test would have caught it and review only just did.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes low-level transport state-machine behavior (including streaming parsing) without an automated/hardware-backed test to validate the end-to-end receive/unwind paths.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@marknolan marknolan self-assigned this Sep 18, 2026
marknolan and others added 2 commits September 21, 2026 09:21
processNackFromCommand() unwound the transaction but sent no progress
report, so an operation containing a refused command never reached its
end value. BluetoothProgressReportPerDevice.updateProgress() derives the
counter from the remaining instruction stack size, and
ShimmerDeviceCallbackAdapter only calls finishOperation() when that
counter equals mProgressEndValue - so a single NACK left the operation
running indefinitely, with mOperationUnderway still true and the
application sitting in CONFIGURING/CONNECTING.

This matters most at connect time. With SD sync enabled the firmware
NACKs every command except SET_SD_SYNC_COMMAND and ACK
(ShimBt_isCmdAllowedWhileSdSyncing), which covers the whole driver
initialisation sequence.

The report is sent before the refused instruction is removed, and only
while it is still queued, to match the arithmetic on the ACK path: the
response-wait path already reported when its ACK arrived, so reporting
again there would skip a count. The three timer-driven commands the ACK
path excludes are excluded here too.

Known limitation: a refused command still counts as progress and the
operation completes as SUCCESS. Reporting a partially-refused operation
as FAIL_WITH_WARNING would mean changing the shared OperationState
handling in finishOperation(), which is a wider change than this fix;
eventCommandRefused() remains the hook for an application that needs to
know.

Co-Authored-By: Claude Opus 5 <[email protected]>
The two earlier harnesses both refused a command while sensing, so both
took the mIsStreaming branch of checkForAckOrRespTask - which clears the
instruction stack but leaves the connection up. The ~2 s teardown the
ticket describes is in the other branch, reached only by a refusal while
not streaming, and nothing exercised it. Configuring mid-stream is not a
supported operation either, so those runs were testing a path the
product does not use.

This drives the supported case. With SD sync enabled the firmware
refuses every command except SET_SD_SYNC_COMMAND and ACK
(ShimBt_isCmdAllowedWhileSdSyncing), so the driver's whole
initialisation sequence is NACKed at connect time. On master that ends
in connectionLost() about two seconds in; on this branch the refusals
are recognised and the link stays up.

Nothing is written to the device, so the run is non-destructive - SD
sync is set once in Consensys beforehand and cleared afterwards. The
test reports INCONCLUSIVE rather than a pass if FULLY_INITIALIZED
arrives, which means sync was not actually enabled.

Compiled against ShimmerPCBasicExamples with Gradle 8.10, the version CI
uses. The checked-in wrapper (6.1) cannot configure ShimmerDriverPC at
all - its Shadow plugin requires Gradle 7+ - which is more evidence the
stale wrapper deserves its own ticket.

Co-Authored-By: Claude Opus 5 <[email protected]>
@MAzalya

MAzalya commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

I reviewed this against current master, which now includes #279, #313 and #319. The driver change looks right, but the SD-sync test case doesn't work, so I'm holding off approving until the not-streaming path has been checked on hardware another way.

Checked and fine

  • Merges onto master (3dedf95b) with no conflicts. ShimmerDriver 177/177 and ShimmerDriverPC 54/54 pass, and ShimmerPCBasicExamples compiles (CI doesn't build that one).
  • Both open Copilot threads are fixed in 648897ba: removal is gated on instructionStillQueued, and LiteProtocol has the response-wait branch. They just aren't marked resolved.
  • Link CRC (added after this branch). With CRC on, the firmware appends CRC bytes to a NACK as well (log-and-stream-common shimmer_bt_uart.c:2409-2426). For 0xFE these are C5 56 (ACK's are F4 65). Neither byte can be mistaken for 0x00/0xFE/0xFF, so on the not-streaming path they're ignored, the same way master already ignores an ACK's CRC.

SD sync can't trigger the not-streaming NACK

Enabling SD sync turns the Bluetooth radio off while idle. In shimmer_config.c:1131-1159 (ShimConfig_checkBtModeFromConfig), BtStop() runs when sdSyncEnabled != btInSyncMode, and BT is only powered back on for btSupportEnabled && !sdSyncEnabled. Otherwise it's left to the sync code (ShimSdSync_start) during a sync recording. The syncEnable ^ ShimBt_isCmdAllowedWhileSdSyncing() gate at shimmer_bt_uart.c:840 only applies while the sync code has the radio on, not on a normal host connection.

We ran TestNackNotStreaming from this branch on a Shimmer3R with SD sync enabled. openPort() fails after about 5.6 s with jssc Port busy, which is Windows giving up on a radio that's off. No command reaches the device. The test still printed "MASTER BEHAVIOUR - the link was torn down after the refusal", because it treats any DISCONNECTED as a teardown, even when the port never opened.

So:

  1. The description's claim that "on master a sync-enabled Shimmer cannot be connected to at all — it drops about two seconds in" needs correcting. The connection never starts, on either branch. The other idle-reachable NACKs in the table are still valid, and still a good reason for the change.
  2. TestNackNotStreaming needs another way to get a NACK, and should give no verdict unless the device actually connected.

Suggested replacement for the bench test

Use a command the firmware always refuses but that changes nothing, during a normal connection with SD sync off:

  1. Connect and wait for NOTIFICATION_SHIMMER_FULLY_INITIALIZED (timeout ~20 s). If it doesn't arrive, report INCONCLUSIVE: could not connect rather than a verdict.
  2. Queue two instructions back to back:
    • SET_CHARGE_STATUS_LED_COMMAND (0x30) with one arg byte 0x00. It's in the 1-arg group (shimmer_bt_uart.c:684) and unconditionally NACKed (shimmer_bt_uart.c:1401), so nothing is written to the device.
    • GET_SAMPLING_RATE_COMMAND (0x03), which proves the stack advances past the refusal (the case Copilot raised).
  3. Observe for 5 s, well past the 2 s ACK_TIMER_DURATION.
    • master: the NACK is ignored, the GET never goes out, and the ACK timer fires → CONNECTION_LOST.
    • This branch: NACK Received for Command: is logged, the refused instruction is dropped, SAMPLING_RATE_RESPONSE arrives, and the device stays connected.
  4. Verdict:
    • PASS: still connected, NACK logged, sampling-rate response received.
    • FAIL: CONNECTION_LOST / DISCONNECTED after the refused command was sent.
    • INCONCLUSIVE: never connected, or the device ACKed 0x30.

0x30 (and SET_BT_COMMS_BAUD_RATE, 0x6A, as an alternative) only NACK from DEV-977 (79abbdbb), which shipped in Shimmer3R LogAndStream v1.01.013+ and Shimmer3 v1.01.004+. Older firmware ACKs them, hence the INCONCLUSIVE case.

The driver has no public writer for these, so the test can queue the bytes the way API_00028_IoThreadTeardownTest does, from the com.shimmerresearch.bluetooth package:

// after FULLY_INITIALIZED
device.getListofInstructions().add(new byte[] {0x30, 0x00}); // SET_CHARGE_STATUS_LED: always NACKed
device.getListofInstructions().add(new byte[] {ShimmerObject.GET_SAMPLING_RATE_COMMAND});

Follow-up (non-blocking)

On the streaming branch (data packet followed by NACK), clearBuffers() doesn't wait for or consume the NACK's CRC bytes. With CRC on, C5 56 then arrives as the next packet's start, which gives one "Unexpected packet header" and possibly one dropped packet before resync. Mirroring #319's bare-ACK handling (isBareAckCrcSoFar / clearBytesFromStartOfBuffers) with the NACK's CRC would close it. Configuring mid-stream is unsupported, so this doesn't need to block the PR.

Once the description is corrected and the reworked test passes on hardware, I'm happy to approve.

Review found the bench test could not work: with SD sync enabled the
firmware switches Bluetooth off while idle, so the port never opened, and
the test still reported "master behaviour" because it took any
DISCONNECTED for a teardown.

It now refuses a command the firmware always refuses and that changes
nothing: 0x30 with one argument byte (the driver's SET_BLINK_LED, the
firmware's SET_CHARGE_STATUS_LED_COMMAND, dispatched but never
implemented), with GET_SAMPLING_RATE queued straight behind it. It follows
those two instructions through the stack and gives a verdict only once
the Shimmer has finished connecting, is idle, and has firmware that
refuses 0x30 (Shimmer3R LogAndStream v1.01.013+, Shimmer3 v1.01.004+):
- PASS: 0x30 retired, the GET answered, the link up 8 s later;
- FAIL: the link torn down, or either instruction never completed;
- INCONCLUSIVE: otherwise.
The window is 8 s because when not streaming the ACK timer gives a SET
ACK_TIMER_DURATION + 3 = 5 s, not 2 s.

It moves to com.shimmerresearch.bluetooth, since the instruction list and
the transaction state are protected, and takes the COM port as its
argument.

On a Shimmer3R, LogAndStream v1.01.025, over classic Bluetooth, 3 runs
each: this branch PASS 3/3 (0x30 retired after 137-224 ms, the GET
answered after 241-272 ms); master FAIL 3/3 (the link lost 5.0 s after
the refusal, no NACK logged).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@marknolan

Copy link
Copy Markdown
Member Author

Thanks. All of it is addressed, and the description is updated.

  • SD sync: you're right that Bluetooth is off while idle with sync enabled. The claim is corrected in the description: the gate only applies while a sync recording has the radio on.
  • Bench test: reworked in 7351b7e as you suggested. It sends 0x30 with a 0x00 argument and queues GET_SAMPLING_RATE behind it, then follows both through the instruction stack. It reports INCONCLUSIVE unless the Shimmer connected, is idle, and has firmware that refuses 0x30.
    • One correction to the plan: when not streaming, a SET gets ACK_TIMER_DURATION + 3 s, so master's teardown comes about 5 s after the refusal, not 2 s. The window is 8 s.
    • Results, on a Shimmer3R on LogAndStream v1.01.025, 3 runs each with the same test source:
      • this branch: PASS 3/3. 0x30 was retired after 137-224 ms and the GET answered after 241-272 ms.
      • master: FAIL 3/3. The link was lost 5.0 s after the refusal, with no NACK logged.
  • Streaming CRC follow-up: done in DEV-1431: a log start the device refuses (NACK) ends at once, idle or mid-stream #345, now stacked on this PR, along with in-memory tests of both NACK paths.
  • Housekeeping: master is merged in with no conflicts, and CI is green. The LiteProtocol Copilot thread is resolved; the fix is in 648897b, as you said.

A side note: 0x30 is the driver's SET_BLINK_LED, and writeLEDCommand() still sends it, including as the check-alive ping while a stream has stalled. So current firmware refuses the driver's own ping.

@JongChern JongChern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The idle path is right: a refusal unwinds the transaction, drops the instruction only while it is still queued, reports progress so an operation still completes, and keeps the link up. The hardware run (3/3 here, 3/3 failing on master) covers the check Mas asked for.

Merge order:

  • Merge this and #345 back to back, before any release. This PR's streaming NACK branch is the first version that #345 replaces (it doesn't wait for the NACK's CRC, clears the rest of the buffer, and fails a command with none in flight).
  • Retarget #345 to master before deleting this branch, or GitHub closes #345.
  • Then the Java HIL at master, the Java release, the Android driver bump with ShimmerResearch/ShimmerAndroidAPI#153, and only then ShimmerResearch/log-and-stream-common#228 out of draft.

Not blocking: a NACK that arrives while a stream has stalled (no data packet before it) still lands at the start of the buffer and isn't handled. Master behaves the same.

@JongChern
JongChern merged commit 1dd2599 into master Nov 13, 2026
2 checks passed
@JongChern
JongChern deleted the DEV-982_nack_handling branch November 13, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants